kourouklidesfandomcom-20200213-history
Artificial Neural Network
This page contains resources about Artificial Neural Networks. For temporal (Time Series) and atemporal Sequential Data, please check Linear Dynamical Systems. Subfields and Concepts * Feedforward Neural Network ** Single-Layer Perceptron (i.e. with no hidden layers) ** Multi-Layer Perceptron (MLP) / Standard Neural Network ** Radial Basis Function (RBF) Network ** Extreme Learning Machine (ELM) ** Convolutional Neural Network (CNN or ConvNet) ** Capsule Network (CapsNet) * Recurrent Neural Network (RNN) ** Hopfield Network ** Boltzmann Machine ** Bidirectional RNN ** Bidirectional associative memory (BAM) ** Long short-term memory (LSTM) ** Gated Rectified Unit RNN (GRU-RNN) ** Simple Recurrent Network (SRN) ** Continuous Time RNN (CTRNN) ** RNN-RBM ** Echo State Network (ESN) ** Unitary RNN (uRNN) * Stochastic Neural Network (i.e. with stochastic transfer function and units or stochastic weights) ** Helmholtz Machine ** Boltzmann Machine ** Restricted Boltzmann Machine (RBM) ** Conditional RBM (CRBM) ** Autoassociative memory ** Generative Stochastic Network ** Generative Adversarial Network ** Stochastic Feedforward Neural Network (with both stochastic and'' deterministic'' hidden units) ** Stochastic Computation Graph ** Variational Autoencoder (VAE) ** Natural-Parameter Network ** Variance Network * Kohonen Network / Self-organizing map (SOM) / Self-organising feature map (SOFM) * Probabilistic Nerual Network ** Bayesian Neural Network (i.e. a Gaussian Process with finitely many weights) *** Probabilistic Backpropagation *** Bayes by Backprop ** Bayesian Dark Knowledge (BDK) ** Natural-Parameter Network (NPN) (i.e. distributions for both the weights and the neurons) *** Gamma NPN *** Gaussian NPN *** Poisson NPN * Random Neural Network * Autoencoder (used for Dimensionality Reduction) ** Linear Autoencoder (equivalent to PCA) ** Stacked Denoising Autoencoder ** Generalized Denoising Autoencoder ** Sparse Autoencoder ** Contractive Autoencoder (CAE) ** Variational Autoencoder (VAE) * Deep Neural Network (i.e. more than two hidden layers) ** Deep Multi-Layer Perceptron ** Deep Belief Network (DBN) ** Convolutional Deep Neural Network ** Long short-term memory (LSTM) ** Deep Autoencoder (i.e. two symmetrical DBN) ** Neural Module Network (NMN) * HyperNetwork ** HyperLSTM * Training ** Automatic Differentiation *** Backpropagation Algorithm *** Backpropagation Through Time (for training RNNs) *** Stochastic Backpropagation ** Optimization *** Stochastic Gradient Methods **** Stochastic Gradient Descent (SGD) **** SGD with Momentum *** Simulated Annealing *** Genetic Algorithms (for training RNNs) ** Contrastive Divergent (CD) Algorithm (for training RBMs) *** Persistent CD (PCD) ** Wake-Sleep Algorithm (for Stochastic ANNs) ** Generative Stochastic Networks (GSN) for probabilistic models ** Auto-Encoding Variational Bayes (AEVB) Algorithm * Activation Functions / Transfer Functions for deterministic units (must be differentiable) ** Logistic ** Rectifier (ReLU) ** Softmax ** Hyperbolic tangent ** Swish * Cost Functions / Loss Functions / Objective Functions ** Least-Squares ** Cross-entropy ** Relative Entropy / KL Divergence ** Connectionist Temporal Classification (CTC) * Energy-Based Model (EBM) ** Free energy (i.e. the contrastive term) ** Regularization term ** Loss Functionals / Loss Functions *** Energy Loss *** Generalized Perceptron Loss *** Generalized Margin Losses *** Negative Log-Likelihood Loss * Improve Generalization (to prevent overfitting) ** Early stopping ** Regularization / Weight decay *** L1-regularization / Laplace prior *** L2-regularization / Gaussian prior *** Max norm constraints ** Dropout ** Add noise * Theory of ANNs ** Representation Theorem ** Universal Approximation Theorem ** Universal Turing Machine Online Courses Video Lectures *Neural Networks for Machine Learning by Geoffrey Hinton - Coursera *Neural networks class by Hugo Larochelle (Youtube ) *Deep Learning and Neural Networks by Kevin Duh Lecture Notes *Convolutional Neural Networks for Visual Recognition by Fei-Fei Li & Andrej Karpathy * CSC321: Introduction to Neural Networks and Machine Learning by Tijmen Tieleman - this might be a bit advanced for beginners * CSC2515: Introduction to Machine Learning by Geoffrey Hinton - this is very similar to the above * Neural Networks and Pattern Recognition by Ömer Cengiz ÇELEBİ Books and Book Chapters See Deep Learning Books. Scholarly Articles * Hannun, A. (2017). Sequence Modeling with CTC. Distill, 2(11). * Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2015). Automatic differentiation in machine learning: a survey. arXiv preprint arXiv:1502.05767. * Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929-1958. * Jacobsson, H. (2005). Rule extraction from recurrent neural networks: A taxonomy and review. Neural Computation, 17(6), 1223-1263. Tutorials Software See Deep Learning Software. See also * Probabilistic Graphical Models * Online Learning Other Resources General * artificial_neural_networks (Github) * What is the difference between CNNs, RBMs and Autoencoders - Stackexchange * Software Tools for RL, ANNs and Robotics - Python and MATLAB * Neural Networks and Deep Learning - free online book * Feedforward Neural Network - Metacademy * Backpropagation - Metacademy * Weight decay in Neural Networks - Metacademy * The Neural Network Zoo - Blog post with some of the ANN architectures * Neural Nets, Connectionism, Perceptrons, etc. - Notebook * Neural Networks, Manifolds and Topology - Blog post * Bayesian Neural Network in PyMC3 (and Theano) - Blog post * Bayesian Neural Network in Edward (and Tensorflow) - Blog post * Hierarchical Bayesian ANN in PyMC3 (and Theano Lasagne) - Blog post * 10 Misconceptions about Neural Networks - Blog post * Artificial Neural Networks Explained - blog post * Pruning deep neural networks to make them fast and small - blog post * Neural Networks gone wild! They can sample from discrete distributions now! - blog post * keras-surgeon - Github * tensorpack - Github * Sequence Modeling With CTC - blog post * gumbel-softmax (Github) - code * bayesgan (Github) - code * Detectron - code * uTensor(Github) - AI inference library based on Mbed and TensorFlow * When is L2 weight decay better than L1 and when is max norm better? - Quora TensorFlow * tensorflow-tutorial - Github * TensorFlow-Tutorials (Hvass-Labs) - Github *TensorFlow-Tutorials (nlintz) - Github * TensorFlow-Examples - Github * stanford-tensorflow-tutorials - Github * tensorlayer - Github * TensorFlow-Book - Github * Tensorflow-101 - Github * tensorflow-generative-model-collection - Github RNN * Recurrent Neural Network - Metacademy * Advanced Recurrent Neural Networks - blog post * Recurrent Neural Networks for Language Modeling - blog post * Awesome-RNN (Github) - A curated list of resources dedicated to RNN * Predict Stock Prices Using RNN (Part 1, Part 2) - blog post * Introduction to LSTMs with TensorFlow - Blog post * Understanding LSTM Networks - Blog post * Understanding LSTM Networks by Example using Torch - Blog post * Time Series Forecasting with the Long Short-Term Memory Network in Python - blog post * Multi-step Time Series Forecasting with Long Short-Term Memory Networks in Python - blog post * How to Seed State for LSTMs for Time Series Forecasting in Python - blog post * Multivariate Time Series Forecasting with LSTMs in Keras - blog post * Unfolding RNNs (Part 1, Part 2) - blog post * LSTM implementation explained - blog post * Time Series Prediction Using LSTM Deep Neural Networks - blog post * Stock Market Predictions with LSTM in Python - blog post * Stock prediction LSTM using Keras (Kaggle) * Predict stock prices with LSTM (Kaggle) * New-York-Stock-Exchange-Predictions-RNN-LSTM (GitHub) - code * Vanilla Recurrent Neural Networks - blog post * How (not) to use Machine Learning for time series forecasting: Avoiding the pitfalls - blog post * Time Series Prediction with LSTM Recurrent Neural Networks in Python with Keras Category:Machine Learning